Natural media painting using a realistic brush and tablet stylus gestures

ABSTRACT

Systems and methods for providing a natural media painting application may receive user inputs through tablet stylus gestures. A user interface may detect stylus gestures that mimic real-world actions of artists based on information collected during user manipulation of the stylus, and may map the gestures to various digital painting and image editing tasks that may be invoked and/or controlled using the gesture-based inputs. The collected information may include spatial and/or directional information, acceleration data, an initial and/or ending position of the stylus, an initial and/or ending orientation of the stylus, and/or pressure data. The stylus gestures may include translations, rotations, twisting motions, mashing gestures, or jerking motions. The application may perform appropriate painting and image editing actions in response to detecting and recognizing the stylus gestures, and the actions taken may be dependent on the work mode and/or context of the graphics application in which stylus gesture was performed.

PRIORITY INFORMATION

This application claims benefit of priority of U.S. ProvisionalApplication Ser. No. 61/224,788 entitled “Methods and Apparatus forNatural Media Painting Using a Realistic Brush and Tablet StylusGestures” filed Jul. 10, 2009, the content of which is incorporated byreference herein in its entirety.

BACKGROUND Description of the Related Art

Natural media painting simulation refers to digital, computer-basedcreation techniques for creating digital paintings or other digitalworks that attempt to mimic real-world “brush, palette, and canvas”painting. Advances in natural media painting simulation have extendedthe conventional digital painting paradigm in ways that have significantpainting workflow implications. New features have added significantcomplexity to the painting workflow. For example, the user may have todeal with cleaning a dirty brush, refilling it when it runs out ofpaint, drying it out to create dry brush strokes, splitting the tip tocreate scratchy strokes, reforming the tip to create fine lines, and soon. Conventional user interface (UI) methodologies require creating UIwidgets to control each of these functions, which forces the user toconstantly switch between focusing on their art on the canvas, andfocusing on widgets on a UI panel. With the size of art canvases quicklyoutpacing the size of monitors, these UI widgets often exist off-screenand the user must first call up the panels before manipulating anddismissing them. All of this amounts to a severe distraction from theactual painting task.

SUMMARY

Various embodiments of methods and apparatus for natural media paintingusing a realistic brush model and tablet stylus gestures are described.Some embodiments may provide methods for performing various digitalpainting and image editing tasks using a natural, gesture-based approachvia a tablet type device and a stylus. Some embodiments may providedetection of stylus gestures that mimic the real-world actions ofartists in real (as opposed to digital) painting, and may performappropriate digital painting and image editing actions in response todetecting one or more of the stylus gestures. By supporting the use ofstylus gestures that mimic the real-world actions, the system may lowerthe cognitive load of the user and allow the user to focus on creativityas opposed to the workings of and user interfaces provided byconventional graphics applications. Some embodiments may enablegesture-based natural media painting workflows by providing a set of sixdegrees of freedom (6DOF) stylus-based gestures for use with stylus andtablet input technologies, which may be augmented with additionalhardware, and by mapping these stylus gestures to painting tasks in anatural media model.

Using a stylus, tablet, and software such as a tablet/stylus inputmodule of a graphics application, embodiments may collect data from thestylus and/or tablet in response to user manipulation of the stylusand/or tablet to, for example, perform various user manipulationtracking tasks including but not limited to acceleration, position,orientation, and proximity detection for the stylus, and touch andpressure detection for the tablet. The collected data may be used torecognize various stylus gestures in real-time or near-real time; therecognized stylus gestures may be mapped to appropriate real-worldpainting actions that are then simulated in the graphics application asdescribed herein.

Stylus gestures that may be detected in embodiments may include, but arenot limited to: translations in one or more dimensions, rotations(including barrel rotations involving twisting about the major axis ofthe stylus), fanning motions (in which the tip of the stylus is wavedback and forth above the tablet), a mashing down motion (i.e. pressingthe stylus into the tablet with high pressure), a jerking up motion(i.e. a quick motion away from the tablet), a jerking down motion (i.e.a quick motion towards the tablet), shaking the stylus away from thetablet (e.g., performed by holding the stylus by its end and flickingthe wrist), and shaking the stylus toward the tablet (e.g., performed byholding stylus by its end and flicking the wrist).

Painting actions performed by a brush tool of the graphics applicationand that may be controlled by various stylus gestures may include, butare not limited to: sharpening the point of a brush by twisting thestylus, cleaning the brush by shaking the stylus away from the tablet,refilling the brush by jerking the stylus down into an area representinga paint well, changing a tool by jerking the stylus up, drying the brushby applying a fanning motion to the stylus, splattering paint from thebrush onto an image by shaking the stylus toward the tablet, addingpaint to the brush by tapping the stylus in an area representing a paintwell, saturating the brush by mashing the stylus down into an arearepresenting a paint well, homogenizing the paint colors on the tip of abrush by twisting the stylus, and splitting the bristles on the tip of abrush by mashing the stylus down onto the tablet.

The stylus gestures and their mapping to natural media painting actionsas provided by various embodiments of the systems and methods describedherein may provide advantages over conventional widget-based userinterfaces for natural media painting for at least the reason that thestylus gestures and their applications in natural media painting mayreduce the amount of time a user needs to execute workflow tasks (e.g.,due to less hand travel, fewer clicks, etc.). In some embodiments, theuse of such gestures may also reduce the cognitive load of the user forthose tasks, since the gestures are aligned with pre-existing motorskills cultivated by traditional painting techniques, and since thegestures can be executed in-place with the artwork, which alleviates theneed to shift focus away from the art to manipulate widgets.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a graphics workstation, accordingto some embodiments.

FIG. 2 is a block diagram illustrating a display on which a userinterface to a graphics application may be implemented, according tosome embodiments.

FIG. 3 is a flow diagram illustrating one embodiment of a method forperforming operations in a natural media painting application usinggestures made with a stylus.

FIG. 4 is a flow diagram illustrating one embodiment of a computerdevice that implements a natural media painting application employingtablet and/or stylus gestures, as described herein.

FIG. 5A illustrates various components of an example stylus, accordingto some embodiments.

FIG. 5B illustrates a brush model, according to some embodiments.

FIGS. 6A-6G illustrate various stylus gestures that may be recognized byan interface module of a natural media painting application, accordingto various embodiments.

FIG. 7 is a flow diagram illustrating one embodiment of a method forcollecting information used to determine a stylus gesture and map thegesture to a particular action to be taken in a natural media paintingapplication.

FIG. 8 illustrates various components of tablet input device, accordingto some embodiments.

FIG. 9 is a flow diagram illustrating one embodiment of a method forperforming proximity-based actions in a natural media paintingapplication.

FIG. 10 is a flow diagram illustrating one embodiment of a method forperforming a proximity-based zoom function in a natural media paintingapplication.

FIG. 11 is a flow diagram illustrating one embodiment of a method forperforming a proximity-based panning function in a natural mediapainting application.

FIGS. 12A-12B illustrate a proximity-based zooming operation in anatural media painting application, according to some embodiments.

FIGS. 13A-13B illustrate a proximity-based panning operation in anatural media painting application, according to some embodiments.

FIG. 14 is a flow diagram illustrating one embodiment of a method forperforming various actions in a natural media painting application usingtouch-and-stylus combination gestures.

FIGS. 15A-15B illustrate examples of touch-and-stylus gestures,according to some embodiments.

FIG. 16 is a flow diagram illustrating one embodiment of a method forusing a touch-and-stylus gesture to assist in performing a paintingoperation in a natural media painting application.

FIGS. 17A-17B illustrate examples of touch-and-stylus gestures that maybe used to assist in performing a painting operation, according to someembodiments.

FIG. 18 is a block diagram illustrating an example computer system thatmay be used to implement the natural media painting application methodsdescribed herein, according to some embodiments.

While various embodiments are described herein by way of example forseveral embodiments and illustrative drawings, those skilled in the artwill recognize that embodiments are not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit the embodimentsto the particular form disclosed, but on the contrary, the intention isto cover all modifications, equivalents and alternatives falling withinthe spirit and scope of the disclosure. The headings used herein are fororganizational purposes only and are not meant to be used to limit thescope of the description. As used throughout this application, the word“may” is used in a permissive sense (i.e., meaning having the potentialto), rather than the mandatory sense (i.e., meaning must). Similarly,the words “include”, “including”, and “includes” mean including, but notlimited to.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following detailed description, numerous specific details are setforth to provide a thorough understanding of claimed subject matter.However, it will be understood by those skilled in the art that claimedsubject matter may be practiced without these specific details. In otherinstances, methods, apparatuses or systems that would be known by one ofordinary skill have not been described in detail so as not to obscureclaimed subject matter.

Some portions of the detailed description which follow are presented interms of algorithms or symbolic representations of operations on binarydigital signals stored within a memory of a specific apparatus orspecial purpose computing device or platform. In the context of thisparticular specification, the term specific apparatus or the likeincludes a general purpose computer once it is programmed to performparticular functions pursuant to instructions from program software.Algorithmic descriptions or symbolic representations are examples oftechniques used by those of ordinary skill in the signal processing orrelated arts to convey the substance of their work to others skilled inthe art. An algorithm as described herein, is generally considered to bea self-consistent sequence of operations or similar signal processingleading to a desired result. In this context, operations or processinginvolve physical manipulation of physical quantities. Typically,although not necessarily, such quantities may take the form ofelectrical or magnetic signals capable of being stored, transferred,combined, compared or otherwise manipulated. It has proven convenient attimes, principally for reasons of common usage, to refer to such signalsas bits, data, values, elements, symbols, characters, terms, numbers,numerals or the like. It should be understood, however, that all ofthese or similar terms are to be associated with appropriate physicalquantities and are merely convenient labels. Unless specifically statedotherwise, as apparent from the following discussion, it is appreciatedthat throughout this specification discussions utilizing terms such as“processing,” “computing,” “calculating,” “determining” or the likerefer to actions or processes of a specific apparatus, such as a specialpurpose computer or a similar special purpose electronic computingdevice. In the context of this specification, therefore, a specialpurpose computer or a similar special purpose electronic computingdevice is capable of manipulating or transforming signals, typicallyrepresented as physical electronic or magnetic quantities withinmemories, registers, or other information storage devices, transmissiondevices, or display devices of the special purpose computer or similarspecial purpose electronic computing device.

Various embodiments of methods and apparatus for natural media paintingusing a realistic brush and tablet stylus gestures are described.Various embodiments may provide methods for performing various digitalpainting and image editing tasks using a natural, gesture-based approachvia a tablet, stylus, and software such as the tablet/stylus inputmodule of the graphics application described herein. Some embodimentsmay provide detection of stylus gestures that mimic the real-worldactions of artists in real (as opposed to digital) painting, and mayperform appropriate digital painting and image editing actions inresponse to detecting the stylus gestures. This may in some embodimentslower the cognitive load of the user, and allow the user to focus oncreativity as opposed to the workings of and user interfaces provided byconventional graphics applications. Various embodiments may enablegesture-based natural media painting workflows by providing a set of sixdegrees of freedom (6DOF) stylus-based gestures for use with a stylusand tablet input technologies, which may be augmented with additionalhardware, and by mapping these stylus gestures to painting tasks in anatural media model.

Advances in natural media painting simulation have extended thetraditional digital painting paradigm in ways that have significantpainting workflow implications. The new features include, but are notlimited to:

-   -   A bristle-based brush model, in which the brush consists of a        set of bristles that dynamically change shape in response to the        physics of the brush stroke. This is in contrast to the single        2D grayscale stamp that is used by conventional digital painting        programs.    -   Support for “wet” and/or “dirty” paint, i.e. support for        bidirectional paint transfer (e.g., from the brush to the        canvas, and from the canvas to the brush), which enables color        blending and smudging in a way that mimics natural paint media.        This is in contrast to a unidirectional paint transfer (i.e. the        transfer of paint from brush to canvas only, without dirtying        the brush) that is used in conventional digital painting        programs.    -   Watercolor painting simulation, which creates the effects of a        brush wet with watery paint that slowly dries during a stroke.        Conventional paint programs do not simulate these secondary        effects, although some use additional tools and textures to        create similar results.

These new features add significant complexity to the painting workflow.Therefore, the user has to deal with cleaning a dirty brush, refillingit when it runs out of paint, drying it out to create dry brush strokes,splitting the tip to create scratchy strokes, reforming the tip tocreate fine lines, and so on. Conventional user interface (UI)methodologies require creating UI widgets to control each of thesefunctions, and force the user to constantly switch between focusing ontheir art on the canvas, and focusing on the widgets on a UI panel. Withthe size of art canvases quickly outpacing the size of monitors, theseUI widgets often exist off-screen and the user must first call up thepanels before manipulating and dismissing them. All of this amounts to asevere distraction from the painting task. However, many of these taskshave corresponding counterparts in the traditional workflow that arecommonly executed by real painters without interrupting their focus onthe canvas. The systems and methods described herein may mimic thesereal-world gestures, and thus may save considerable time for users ofnatural media painting workflows.

As described above, the natural media painting model has workflowimplications that may benefit from stylus-based gesture input. In someembodiments, by providing stylus gestures that may mimic real-worldworkflows of real painting, the systems and methods described herein maylower the cognitive load of the user and allow the user to focus oncreativity as opposed to the workings of and user interface to aconventional graphics application. Some embodiments may enablegesture-based natural media painting workflows by providing a set of sixdegrees of freedom (6DOF) stylus-based gestures for use with tabletinput technologies which may be augmented with additional hardware, andby mapping these stylus gestures to painting tasks in a natural mediamodel.

Using a stylus, tablet, and software such as a tablet/stylus inputmodule of a graphics application, such as that illustrated in FIG. 1,some embodiments may collect data from the stylus and/or tablet inresponse to user manipulation of the stylus and/or tablet to performvarious user manipulation tracking tasks. The collected data mayinclude, but is not limited to, acceleration, position, orientation, andproximity data detected for or by the stylus, and touch and pressuredata detected for or by the tablet. The collected data may be used torecognize various stylus gestures in real-time or near-real time, andthe recognized stylus gestures may be mapped to appropriate real-worldpainting actions that are then simulated in the graphics application asdescribed below.

FIG. 1 illustrates an example graphics workstation configured toimplement the systems and methods described herein, according to variousembodiments. As illustrated in this example, the workstation mayinclude, but is not limited to, a computer device 100, one or moredisplays 102, a tablet input device 130, and a stylus 140. An examplecomputer device which may be used in some embodiments is furtherillustrated in FIG. 4. As illustrated in this example, computer device100 may implement a graphics application 120, which may be a naturalmedia painting application, as described herein. Graphics application120 may include a bristle brush model 123, a painting simulation module127, a tablet/stylus input module 122, and/or a gesture mapping module125. In some embodiments, bristle brush model 123 may be a component ofpainting simulation module 127.

Graphics application 120 may provide a user interface (UI) 124 via oneor more displays 102. Graphics application 120 may display, for examplein a window provided by the UI 124 on the one or more displays 102, animage 126 that a user is currently working on (e.g., either creating orediting). Graphics application 120 may provide a painting or drawingtool 128 that the user may manipulate, for example via tablet 130 and/orstylus 140, to create or edit content in image 126. The tool 128 may,for example, have various modes that emulate a paintbrush, pencil,eraser, spray can, and so on. While embodiments are generally describedas providing gesture-based manipulations of a paintbrush tool, it is tobe noted that similar techniques may be applied to other types ofpainting or drawing tools.

Stylus 140 may be configured to be held in a hand of the user and to bemanipulated by the user in relation to tablet 130 to perform variousimage editing operations or other tasks. The user may manipulate stylus140 and/or tablet 130 in various ways. For example the user may movestylus 140 away from tablet 130 or towards tablet 130; move stylus 140up and down, left and right, or diagonally and so on in relation totablet 130; rotate stylus 140 on one or more axes; touch a pressuresensitive surface of tablet 130 with stylus 140 and/or with a finger,knuckle, fingernail, etc.; apply varying amounts of pressure to thetouch and pressure sensitive surface of tablet 130 with a finger orstylus 140; move the tip of stylus 140 on the touch and pressuresensitive surface of tablet 130; and so on. Tablet 130 is configured todetect the various manipulations performed by the user with stylus 140and/or with a finger, knuckle, etc. on the surface of tablet 130 andcommunicate information regarding the manipulations to tablet/stylusinput module 122 on computer device 100, for example via a wired orwireless interface.

Tablet/stylus input module 122 may be implemented as a component ormodule of application 120, as a library function, as a driver, or assome other software entity. Tablet/stylus input module 122 may beimplemented in software, in hardware, or as a combination of hardwareand software. Graphics application 120, via tablet/stylus input module122, may interpret the information regarding the manipulations to detectvarious gestures and to perform various painting actions in response tothe detected gestures for creating or editing content of image 126. Forat least some of those actions, painting tool 128 may be appropriatelymoved, modified, and/or otherwise affected on display 102. Variousexamples of gestures that may be detected are listed below, as arevarious examples of painting actions that may be invoked and/orcontrolled by such stylus gestures.

In some embodiments, software and/or hardware on tablet 130 may performat least some of the functionality of detecting various gestures. Thus,in some embodiments, tablet 130 may be configured to detect gestures andcommunicate the detected gestures to graphics application 120, whichthen performs the appropriate painting actions in response to thegestures. In other embodiments, tablet 130 may only collect informationregarding gestures and communicate the gestures to application 120 viatablet/stylus input module 122; tablet/stylus input module 122 mayperform the function of detecting the gestures from the information andcommunicating the gestures to application 120, or to other modules ofapplication 120, which then performs the appropriate painting actions inresponse to the gestures.

FIG. 2 illustrates an example display 200 on which a user interface to agraphics editing module, such as image editing operations module ofgraphics application 120 may be implemented, according to oneembodiment. In this example, the display is divided into four regions orareas: menus 210, tools 202 (which may include a “fill” tool, a “clean”tool, and/or a mode selection tool), controls 204 (which may includepalette 206), and work area 208. Tools 202 may include one or moreuser-selectable user interface elements. In this example, it is thisarea that contains the user interface elements that a user may select toapply various effects to the image. For example, the user may select abrush tool for use in applying paint to an image being created and/oredited in work area 208. Other optional tools may be selected as well,such as an eraser or reset function, in some embodiments. While FIG. 2shows the elements in tools area 202 as buttons, other types of userinterface elements, such as pop-up or pull-down menus, may be used toselect from among one or more tools in various embodiments. For example,in one embodiment, the mode selection mechanism illustrated in toolsarea 202 may be implemented using a pop-up or pull-down menu to select awork mode, such as an automatic zoom mode or an automatic panning mode.As noted above, the reset and eraser tools are optional, and thus may ormay not be included on the user interface in various embodiments.Various embodiments may include other tools not shown as well, such asan “undo” tool that undoes the most recent user action in the work area208.

In this example, controls 204 may include one or more user-modifiablecontrols, such as slider bars, dials, pop-up menus, alphanumeric textentry boxes, etc., for specifying various parameters of the paintingfunctions to be applied to an image (e.g., using the brush tool). Inthis example, two slider bars are provided to specify different values(or relative values) of configurable parameters of a painting function.For example, slider bars may be used to specify an amount of ink, apigment concentration amount, a transparency value, a brush width, abristle stiffness, or other parameters that are to be applied when usingthe brush tool to “paint” on the image being created or edited in workarea 208. Various methods of specifying values of any of the otherparameters used in simulating painting effects (i.e. methods other thanthose illustrated in FIG. 2) may be used in other embodiments. In someembodiments, slider bars or another input mechanism in controls area 204may be used to specify one or more threshold distance values for usewith proximity based gestures and their corresponding functions in thegraphics application, such as those described in more detail below. Insome embodiments, slider bars or another input mechanism in controlsarea 204 may be used to specify a zoom level for an automated zoomfunction or to override a default zoom level for such a function.

In the example illustrated in FIG. 2, menus 210 may include one or moremenus, for example menus used to navigate to other displays in thegraphics application, open files, print or save files, undo/redoactions, and so on. In this example, work area 208 is the area in whichan image being created or edited is displayed as graphics editingoperations are performed. In various embodiments, work area 208 maydisplay a portion or all of a new image to which paint is to be added,or a portion or all of a previously existing image being modified byadding paint, as described herein. In the example illustrated in FIG. 2,work area 208 illustrates an image in progress.

Realistic simulation of natural media painting in a digital medium iscompelling because it enables a wide variety of effects that aredifficult to achieve with conventional digital tools. However, realpaint media may be more difficult to manipulate than digital tools, andmay take a considerable amount of time and practice to master.Embodiments of the painting simulation module described herein mayleverage capabilities of digital tools and techniques to ease theworkflows and learning curves associated with natural media paints. Insome embodiments, various methods for brush cleaning, brush filling, andapplying paint strokes in a manner that mimics real world paintingactions may be provided to enable common natural media paint tasks to beautomated in a digital painting tool, but with a fairly realistic userexperience.

Some embodiments of a painting simulation module, such as paintingsimulation module 127 described herein may employ a bristle brush model(such as bristle brush model 123) that simulates a brush tip and thenotion of the paint being loaded onto the brush tip and updated duringstrokes and when the painting color is set. The user may also perform a“clean” action, which is equivalent to dipping the brush in, forexample, turpentine or water, to remove the paint load, and a “fill”action, which is equivalent to dipping the brush into a well or paint ona palette, replenishing the paint on the brush. In some embodiments,during the act of stroking with the brush, the brush's paint load willbe depleted, eventually running out, and the brush may dirty, picking uppaint from the canvas, as with a real brush. The clean and fill actionsmay allow the user to manage the paint load between strokes for thedesired stroke effect.

Some embodiments of a painting simulation module may provide an “autoclean” option and/or an “auto fill” option. When turned on or selected,these options may trigger automatic clean and/or fill actions betweenstrokes, for example at the beginning of each new stroke. In otherembodiments, a user interface module may provide an interface throughwhich the user may manually fill or clean the brush between strokes. Insuch embodiments, the user may manually control cleaning and filling ofthe brush as necessary or desired, similar to the way a painter woulduse turpentine/water and a paint palette to clean and load a physicalpaintbrush. Many types of paint effects may require functions havingthis level of control, such as for depositing some paint to be mixed onthe canvas that will shortly dry out and then be blended with thesurrounding paints (e.g., to add highlights or touches of color), or forperforming multiple stroke tasks with varying brush loads. In someembodiments, to fill a brush with paint, a user may select a “fill”control (such as that illustrated in tools area 202 of FIG. 2), whichmay load paint of a selected color (e.g., a color selected from palette206).

The example user interface illustrated in FIG. 2 provides user interfaceelements to enable or disable an auto clean mode and an auto fill modeaccording to some embodiments. In this example user interface, a “clean”user interface element (shown as a radio button) is provided whereby auser may manually clean a brush if and when desired, or may enable ordisable the auto clean option, in different embodiments. The userinterface illustrated in FIG. 2 also includes a color palette whereby auser may manually load a brush with paint if and when desired, and a“fill” user interface element (shown as a radio button) whereby a usermay enable or disable the auto fill option.

In some embodiments, a tablet/stylus input module may provide aninterface through which the user may fill or clean the brush betweenstrokes by applying particular gestures to a stylus (such as stylus140), and these gestures may mimic the real world actions used to filland clean a physical bristle brush, e.g., pressing the brush into apaint well to fill it, and shaking the brush to clean it.

One embodiment of a method for performing various operations of anatural media painting application using gestures made with a stylus isillustrated in FIG. 3. As illustrated at 310 in this example, the methodmay include a natural media painting application collecting informationabout a user manipulation of a stylus. For example, the application maycollect spatial information collected during the manipulation of thestylus, directional information collected during the manipulation of thestylus, acceleration data collected during the manipulation of thestylus, an initial position of the stylus, an ending position of thestylus, an initial orientation of the stylus, or an ending orientationof the stylus.

The method may include the application determining a stylus gesture,based on the collected information, as is 320. For example, theapplication may determine that the collected information indicates atranslation motion performed by the stylus to move it in one or moredimensions, a twisting motion, a barrel rotation, a fanning motion, amashing down of the stylus into a tablet device, a jerking up motion, ajerking down motion, a shaking of the stylus away from the tabletdevice, or a shaking the stylus toward the tablet device

As illustrated at 330 in FIG. 3, the method may include the applicationdetermining an action to be taken within the application, based on thestylus gesture. For example, one or more painting actions or work modechanges may be mapped (e.g., using gesture mapping module 125 in FIG. 1)to each of a plurality of gestures that can be made using the gestureand detected by the application (e.g., through various interfaces, suchas tablet/stylus input mode 122 in FIG. 1). Once the appropriate actionhas been determined, the method may include performing the determinedaction, as in 340.

In some embodiments, a combination of inputs may be analyzed todetermine an action to take in response to a gesture made by a stylus.These inputs may include information collected about the manipulation ofthe stylus itself and/or other information that may provide a contextfor the gesture, an indication of a workflow in progress, or anindication of a work mode or other parameter of an image editingsession. FIG. 4 illustrates one embodiment of a method for collectinginputs for determining a painting action to be taken in response to auser manipulating a stylus. As illustrated in FIG. 4, the method mayinclude a natural media painting application collecting informationabout a user manipulation of a stylus (as in 410), and determining astylus gesture dependent on the information collected (as in 420).

As described herein, in some embodiments, the natural media paintingapplication may detect and act upon an input indicating that a stylus isin proximity to a tablet input device. As illustrated in FIG. 4, amethod for performing an action in the application may includedetermining the proximity of the stylus with respect to such a device,as in 430. For example, in some embodiments, a work mode (e.g., aviewing mode, display mode, or other configurable work flow mode) may beselected, invoked, or changed based on whether the stylus is within agiven distance of the tablet input device. In such embodiments, a givenstylus gesture may be mapped (e.g., by a gesture mapping module of thenatural media painting application) to one painting action in a givenmode, and another painting action in another mode. Similarly, somestylus gestures may be mapped to different painting actions based on thecontext in which the gesture was made (e.g., whether it was made duringa stroke, between strokes, while zoomed in, while zoomed out, etc.)Various examples of context-sensitive gestures and their mappings topaining actions are described below.

As illustrated in FIG. 4, the method may include the applicationdetermining the context of the stylus gesture (as in 440), and maydetermine the painting action that is mapped to the stylus gesture,dependent on the determined context (as in 450). The application maydetermine the work mode for the painting action (as in 460), based onone or more of the proximity of the stylus with respect to the tablet,one or more previous painting actions, the state of an image beingcreated or edited, the currently selected viewing options, or otherfactors. Various examples of the modes in which a painting action may beapplied are described below. Once the painting action and work mode havebeen determined, the method may include performing the painting actionaccording to the determined mode, as in 470.

FIG. 5A illustrates components of an example stylus 140 according tosome embodiments. Stylus 140 may generally be described as having a tip142 and a handle 144. Note that a stylus 140 may be provided with twotips instead of one as shown in FIG. 5A. Stylus 140 may include one ormore accelerometers 148 and/or other components for sensing movementmetrics including but not limited to spatial (location), directional,and acceleration metrics. This motion information may be communicated toa tablet, such as tablet 130 shown in FIGS. 1 and 2, via an interface146. Interface 146 may typically be a wireless interface, although wiredinterfaces are possible.

In some embodiments, the natural media painting application may employ abrush model that simulates the use of a bristle brush, such as onerepresented by a brush tool that is manipulated using a stylus. In suchembodiments, realistic brush behavior may be simulated in the paintingapplication. In one embodiment, the brush behavior may be simulatedsubstantially in real-time to deposit ink or paint onto a virtualcanvas. A brush model may include a large set of discrete bristles. Thebristles may comprise “virtual” bristles and may also be referred toherein as bristle representations. The simulated behavior of the brushmay result in continuous strokes created by sweeping individual bristlesinto quadrilaterals. The brush model and brush behavior simulation maymodel the change of the shape of a brush tip during a stroking motionand the deposition of paint or ink resulting from the motion. Bycomputing the effect of each bristle independently of other bristles, afaithful reproduction of how a real brush deposits paint or ink andchanges shape during a stroke may be achieved.

FIG. 5B is a block diagram illustrating a simplified brush modelaccording to one embodiment. In one embodiment, a brush may be simulatedas a set of discrete “virtual” bristles. As shown in FIG. 5B, the brushmodel 500 may comprise a plurality of bristles 530. In one embodiment,the brush model may also comprise a brush handle 535. The plurality ofbristles 530 may be attached to the end of the handle 535. In oneembodiment, the bristles 530 may include a sufficient number ofindividual bristles (e.g., 50 to 100) for accurate simulation of thebehavior of a real brush.

At least some of the stylus gestures that may be detected by a naturalmedia painting application, such as the stylus gestures marked with anasterisk (*), may be performed using a stylus augmented with one or moreaccelerometers, and possibly other hardware and/or software, forcollecting motion and other data to be used in gesture recognition.Stylus gestures that may be detected in various embodiments may include,but are not limited to:

-   -   a) A barrel rotation (a twisting motion about the major axis of        the stylus);    -   b) A fanning motion (waving the stylus tip back and forth above        the tablet);    -   c) Mashing down (pressing the stylus into the tablet with high        pressure);    -   d) *A jerk up (a quick motion away from the tablet);    -   e) *A jerk down (a quick motion toward the tablet);    -   f) *Shaking away from the tablet (holding the stylus by its end        and flicking the wrist); and    -   g) *Shaking toward the tablet (holding the stylus its end and        flicking the wrist).

Some of the stylus gestures that may be recognized by an interfacemodule of a natural media painting application, such as tablet/stylusinput module 122 in FIG. 1, are illustrated in FIGS. 6A-6G, according tovarious embodiments. For example, FIG. 6A illustrates varioustranslation gestures in which the stylus moves in a given direction(e.g., up/down, right/left, etc.) FIG. 6B illustrates a barrel rotation(e.g., a rotation about the major axis of the stylus). FIG. 6Cillustrates a fanning motion, as described above. FIG. 6D illustrates ajerk up motion, and FIG. 6E illustrates a jerk down motion. FIGS. 6F and6G illustrate the effects of different stylus gestures on a brush model,such as brush model 500 in FIG. 5B. For example, FIG. 6F illustrates theeffect of a stylus gesture that corresponds to pressing a bristle brushlightly on a canvas, while FIG. 6G illustrates the effect of a stylusgesture that corresponds to mashing a bristle brush down into a canvas,as in(c) above. In general, tablet/stylus input module 122 may beconfigured to recognize a wide variety of stylus gestures by detectingmanipulation of the stylus from an initial pose (e.g., an initialposition and orientation) using six degrees of freedom (e.g., detectingmovement in a given direction in three dimensions, rotation about anaxis in any dimension, pitch, roll, yaw, etc.) As described herein, this6DOF information may be augmented with information collected from anaccelerometer, various proximity sensors, a touch and/or pressuresensitive tablet device, or other input mechanisms to define a stylusgesture that is mapped to an action to be taken in a natural mediapainting application, and the action mapped to the gesture may bedependent on a work mode and/or context in which the stylus gesture wasmade.

FIG. 7 illustrates one embodiment of a method for collecting informationused to determine a stylus gesture and map the gesture to a particularaction to be taken in a natural media painting application. Asillustrated in this example, the method may include a natural mediapainting application determining an initial pose of the stylus,including its position and orientation (as in 710). For example, in someembodiments, the stylus and/or a tablet input device may include sensorsthat allow the initial pose of the stylus to be determined by aninterface module, such as tablet/stylus input module 122. In otherembodiments, the system may employ input from a camera that may beanalyzed to determine the initial pose of the stylus.

If the stylus is in proximity to a tablet input device, shown as thepositive exit from 720, the method may include the applicationdetermining the distance from the stylus to the tablet input device, asin 730. For example, in some embodiments, if the stylus is within agiven distance from the tablet, sensors in the stylus and/or tablet maydetect that the stylus is in proximity to the tablet. In otherembodiments, the system may employ input from a camera that may beanalyzed to determine whether the stylus is in proximity to the tablet.As described herein, some of the functions of the natural media paintingdevice may be dependent on whether the stylus is in proximity to thetablet device or on a change in the proximity of the stylus with respectto the tablet device.

As illustrated at 740 in FIG. 7, the method may include determiningwhether the stylus and/or the user is touching the tablet input device(e.g., using a finger, fingernail, etc.). If so, shown as the positiveexit from 740, the method may include the application collecting touchposition and/or pressure data from the tablet input device for thestylus and/or the user touch throughout the gesture, as in 750. Asillustrated in this example, the method may include the applicationdetecting the motion of the stylus throughout the gesture, andcollecting data related to any or all of a translation (i.e. a movementin a given direction), rotation, pitch, roll, yaw, and acceleration ofthe stylus during the gesture, as in 760.

FIG. 8 illustrates components of an example tablet input deviceaccording to some embodiments. As illustrated in this example, tablet130 may include a touch and pressure-sensitive surface 832 that may beconfigured to detect contact with tip 142 of stylus 140, and/or contactwith another object such as the user's fingertip or knuckle. Surface 832may also be configured to detect motion on the surface, for exampledetecting the dragging of tip 142 of stylus 140 across the surface.Surface 832 may also be configured to detect the amount of pressureapplied to the surface, e.g., by stylus 140, another object, or a usertouch. Tablet 130 may also include an interface to stylus 836 that isconfigured to detect the position of, and motion of, stylus 140 inrelation to tablet 130, for example by receiving input from stylus 140via a wireless interface, or alternatively via one or more motiondetectors integrated in or coupled to tablet 130 that are configured totrack the motion and position of stylus 140. In some embodiments, tablet130 and/or stylus 140 may include a camera, through which input aboutthe position and/or motion of stylus 140 may be collected (not shown),or such a camera may included as an additional component of the systemseparate from tablet 130 and stylus 140. In some embodiments, tablet 130may also include an input processing module 838 configured to processinput received via interface to stylus 836 and/or surface 832.

Input processing module 838 may also include an interface to computerdevice 834. Interface 834 may be a wired or wireless interface.Interface 834 may be configured to communicate information collectedfrom interface 836 and/or surface 832 to a computer device such ascomputer device 100 of FIG. 1. A graphics application on the computerdevice, such as graphics application 120 of FIG. 1, may interpret theinformation to detect various gestures and to perform various paintingactions in response to the detected gestures for creating or editing thecontent of images, as described herein. In some embodiments, inputprocessing module 838 may be configured to perform at least some of thefunctionality of detecting and/or recognizing various gestures. Thus, insome embodiments, tablet 130 may be configured to detect/recognizegestures and communicate the gestures to a graphics application viainterface 834. The graphics application may then perform the appropriatepainting actions in response to the gestures.

As described in more detail herein, the actions taken in response to astylus gesture may include painting actions and/or work flow actions(e.g., changing a parameter of the work flow, work environment, viewingcontext, or another mode of the natural media painting application). Forexample, the painting actions that may be controlled by the stylusgestures in some embodiments may include, but are not limited to:

-   -   a) Sharpening the point of a brush by twisting, as defined for        stylus gesture (a) above. As a brush dries, its tip may become        split, and the stylus may thus no longer make a smooth line        stroke. Sharpening the brush to a point restores the original        shape of the brush.    -   b) Cleaning the brush by shaking it away from tablet, as defined        for stylus gesture (f) above. As a brush is dragged through        paint on the canvas, the tip dirties and the color changes.        Cleaning the brush restores its original loaded color, or        alternately, removes all paint.    -   c) Refilling the brush by jerking it down, as defined for stylus        gesture (e) above. During a stroke, the brush may run out of        paint. Refilling restores the originally loaded color.    -   d) Changing the tool by jerking it up, as defined for stylus        gesture (d) above. Rapid switching between paintbrushes is thus        enabled.    -   e) Drying the brush by fanning, as defined for stylus gesture        (b). With a wet paintbrush, in order to make dry strokes the        brush tip must first be dried. This is the opposite action of        refilling the brush (c), which re-applies paint to the brush        tip.    -   f) Splattering paint by shaking the brush toward tablet, as        defined for stylus gesture (g) above. A wet brush tip will drip        paint to make characteristic spattering effects.    -   g) Adding paint to the brush by tapping it in a paint well. When        mixing colors, the artist may want to pick up some small amount        of paint on the brush to adjust the mixed color.    -   h) Saturating the brush by mashing it in a paint well, as        defined for stylus gesture (c) above. When loading the brush        with paint before stroking, the artist may want to fill the        entire brush tip with the selected color.    -   i) Homogenizing (i.e. mixing or blending) the colors of paint on        the tip of the brush by twisting the brush, as defined for        stylus gesture (a) above. When stroking across paint on the        canvas, the brush tip dirties differently on different parts of        the tip, which can create strokes of non-uniform color.        Homogenizing the tip colors ensures strokes will be a single        color, while retaining the dirty state.    -   j) Splitting the bristles of the brush tip by mashing the brush        down, as defined for stylus gesture (c) above. To create        scratchy strokes, the brush tip must first be split into        separate clumps.

As previously noted, the stylus gestures and the mapping of thesegestures to natural media painting actions, as provided by variousembodiments, may provide advantages over conventional widget-based UIsfor natural media painting for at least the reason that the stylusgestures and their applications in natural media painting may reduce theamount of time a user needs to execute workflow tasks (e.g., due to lesshand travel, fewer clicks, etc.). Some embodiments may also reduce thecognitive load of the user for those tasks, since the gestures arealigned with pre-existing motor skills cultivated by traditionalpainting techniques, and since the gestures can be executed in-placewith the artwork, which alleviates the need to shift focus away from theart to manipulate widgets.

In some embodiments, the system and methods described herein may providegesture-based zooming. In conventional graphics applications, usersgenerally cannot perform detailed, zoomed-in, pixel-level editing of animage, for example at 1:1 ratio, while at the same time displaying theentire image, as the resolution of most display devices is insufficientto display both representations at the same time. Conventionally, usersmay have to zoom in and out on the image, and may have to repeatedlytoggle the screen mode from a windowed mode to a full screen mode. Thus,users typically perform pixel-level editing while zoomed-in on a smallportion of the image displayed on the screen. After editing, the usermust zoom out to view the changes in context of the entire image. Theseconventional pixel-level editing techniques require pushing a button,depressing a key sequence, or other user interactions to zoom in and outthat complicate the workflow. In some embodiments of the systems andmethods described herein, a particular stylus gesture may be mapped to a“zoom in” or “zoom out” function in the natural media paintingapplication, in some contexts.

As noted above, in some embodiments, the system and methods describedherein may perform various functions and/or operate in particular modesbased on the proximity of a stylus with respect to a tablet input, or ona change in the proximity. One embodiment of a method for performingvarious proximity-based actions in a natural media painting applicationis illustrated in FIG. 9. As illustrated at 910, the method may includea natural media painting application determining an initial position ofa stylus with respect to a tablet input device (e.g., using varioussensors, input from a camera, etc). The method may also includedetermining an initial distance from the stylus to the tablet inputdevice, as in 920. For example, in some embodiments, one or more sensorsand/or cameras in the stylus, tablet device, or another component of thesystem may be used to determine the initial distance from the stylus tothe tablet device. In some embodiments one or more sensing mechanisms inthe tablet may detect that the stylus was or has recently moved withinor beyond a given threshold distance. In some embodiments, the systemmay use two or more threshold distances to determine proximity withrespect to the tablet. For example, different distance thresholds may beused in different proximity based gestures and/or their correspondingactions. These proximity distance thresholds may be pre-defined by theapplication, or may be configurable by the user (e.g. through an inputmechanism in the user interface of the application), in differentembodiments.

As illustrated at 930 in this example, the method may includedetermining that the stylus has moved with respect to the tablet inputdevice, e.g., moving closer to or farther away from the tablet. In someembodiments, the change in proximity may be detected even when thestylus does not touch the tablet in either or both of its initialposition or its new position. As illustrated at 940, the method mayinclude the application determining a work mode change that isrepresented by the movement of the stylus with respect to the tablet.For example, a change in the viewing mode or other work flow context maybe mapped to a particular change in the proximity of the stylus, and maybe based on an absolute change (e.g., with the stylus moving closer toor farther away from the tablet) and/or on the rate of change (e.g.,whether the stylus moved quickly or slowly toward or away from thetablet). In response to determining a work mode change that isrepresented by the change in the proximity of the stylus, theapplication may apply the determined work mode change in theapplication, as in 950.

In some embodiments, the system and methods described herein may allowthe use of a tablet and stylus to provide a “zoom when near” or “zoomfor detail” function, referred to herein as a zoom function or auto zoomfunction, that simplifies the workflow for performing detailed editingthat requires zooming into the image. In one embodiment, the auto zoomfunction may be provided as an option in a graphics application and/oras an option on a tablet/stylus. When the function is enabled, as thestylus is moved into proximity of the tablet, the application may zoominto a displayed image to focus on an area of interest; and the user maythen perform appropriate detail edits, for example on individual pixelsof the image. When the stylus is moved away from proximity to thetablet, the application may zoom out on the displayed image to, forexample, fit the entire image on the display. In some embodiments,scaling factors (i.e. zoom levels) for the auto zoom function may befixed. In other embodiments, scaling factors for the auto zoom functionmay be specified or changed via one or more user interface elementsprovided by or for the application, the tablet, and/or the stylus. Insome embodiments, proximity of the stylus to the tablet may bedetermined by comparing a current location of the stylus to apre-specified proximity threshold to at least one position on thetablet; if the current location of the stylus, or a portion of thestylus, is closer than the proximity threshold to the at least oneposition on the tablet, then the stylus is considered to be proximate tothe tablet.

One embodiment of a method for performing a proximity-based zoomfunction in a natural media painting application is illustrated in FIG.10. As illustrated at 1010, the method may include a natural mediapainting application determining an initial position of a stylus withrespect to a tablet input device (e.g., using various sensors, inputfrom a camera, etc). The method may also include determining an initialdistance from the stylus to the tablet input device, as in 1020. Forexample, in some embodiments, one or more sensors and/or cameras in thestylus, tablet device, or another component of the system may be used todetermine the initial distance from the stylus to the tablet device.

As illustrated at 1030 in this example, the method may includedetermining that the stylus has moved with respect to the tablet inputdevice, which may include the stylus moving closer to or farther awayfrom the tablet. In some embodiments, a change in proximity may bedetected even when the stylus does not touch the tablet in either orboth of its initial position or its new position (e.g., using one ormore sensors and/or cameras in the stylus, tablet device, or anothercomponent of the system). As illustrated in this example, if the autozoom feature is not enabled for the current work session, shown as thenegative exit from 1040, there may be no change in the zoom level inresponse to this stylus motion (as in 1070). For example, the auto zoomfeature may in some embodiments be enabled by default. In otherembodiments, this feature may be selectively enabled by the user (e.g.,by selecting an “auto zoom” mode using the mode control mechanism intools area 202 of FIG. 2).

If the auto zoom feature is enabled for the current work session, shownas the positive exit from 1040, the method may include determiningwhether the stylus moved closer to the tablet input device, as in 1050.In this example, if the stylus moved closer to the tablet input device,shown as the positive exit from 1050, the method may include theapplication changing the work mode to zoom in to an area of interest(e.g., a most recent work area, or an area above which the stylus hoverswhen brought closer to the tablet) to perform detailed editing work, asin 1055. If not, shown as the negative exit from 1050, the method mayinclude the application determining whether the stylus moved fartheraway from the tablet, as in 1060. In this example, if the stylus movedfarther away from the tablet input device, shown as the positive exitfrom 1060, the method may include the application changing the work modeto zoom out, e.g., to display a larger context for the paintingoperations in progress, as in 1065. In some embodiments, when zoomingout using such an auto zoom feature, the application may display theentire image that is being created and/or edited. If the stylus has notmoved farther away from the tablet, shown as the negative exit from1060, there may be no change in the zoom level in response to thisstylus motion (as in 1070). In other words, if the stylus has moved withrespect to the tablet, but not in a direction that increased ordecreased its distance to the tablet, there may be no change in the zoomlevel in response.

In addition to the zoom function, some embodiments may providefunctionality that allows the user to pan the image when the stylus isin proximity to, but not touching, the tablet surface. In someembodiments, panning may be performed by motion of the stylus. In otherembodiments, panning may be performed by holding the stylus in proximityto the tablet and manipulating another user input device such as atrackpad, trackball, joystick or mouse. In some embodiments, the imagemay be zoomed to a subwindow surrounding and tracking the stylus to thusprovide a loupe-like function.

One embodiment of a method for performing a proximity-based panningfunction in a natural media painting application is illustrated in FIG.11. As illustrated at 1110, the method may include a natural mediapainting application determining an initial position of a stylus withrespect to a tablet input device (e.g., using various sensors, inputfrom a camera, etc). The method may also include determining an initialdistance from the stylus to the tablet input device, as in 1120. Forexample, in some embodiments, one or more sensors and/or cameras in thestylus, tablet device, or another component of the system may be used todetermine the initial distance from the stylus to the tablet device.

As illustrated at 1130 in this example, the method may includedetermining whether an auto pan feature is enabled for the current worksession. If not, shown as the negative exit from 1130, there may be nopanning performed in response to this stylus motion (as shown in 1180).For example, the auto pan feature may in some embodiments be enabled bydefault. In other embodiments, this feature may be selectively enabledby the user (e.g., by selecting an “auto pan” mode using the modecontrol mechanism in tools area 202 of FIG. 2).

If the auto pan feature is enabled for the current work session, shownas the positive exit from 1130, the method may include the applicationdetermining whether the stylus is in proximity to the tablet device, asin 1140. In other words, the method may include the applicationdetermining if the determined distance between the stylus and the tabletis within a given threshold distance for performing proximity-basedpanning. As previously noted, in some embodiments, a single thresholdmay be defined for the performance of proximity-based actions, while inother embodiments, a different threshold may be defined for each of aplurality of proximity-based actions, or for the used of proximity-basedactions in different contexts. These thresholds may be predefined or maybe configurable by the user (e.g., through a parameter setting mechanismof controls area 204 in FIG. 2), in different embodiments. If the stylusis not in proximity to the tablet device (e.g., according to a thresholdfor proximity-based panning), shown as the negative exit from 1140,there may be no panning performed in response to this stylus motion (asshown in 1180).

If the stylus is in proximity to the tablet device (e.g., according to athreshold for proximity-based panning), shown as the positive exit from1140, the method may include determining that the stylus has moved withrespect to the tablet input device, e.g., moving closer to or fartheraway from the tablet (as in 1150). In some embodiments, a change inproximity may be detected even when the stylus does not touch the tabletin either or both of its initial position or its new position (e.g.,using various sensors, input from a camera, etc.). In some embodiments,an auto pan feature may be used in conjunction with an auto zoomfeature, e.g., to allow a user to zoom and pan to or from a particulararea of interest in one motion. As illustrated in this example, if anauto zoom feature is also enabled for the current work session, shown asthe positive exit from 1160, the method may include the applicationchanging the work mode to pan and zoom according to the stylus motion,as in 1165. For example, if the stylus moved closer to the tablet inputdevice and to the right, the application may change the work mode tozoom in to an area of interest that is centered to the right of the areapreviously centered in the display in order to perform detailed editingwork in that area. If the stylus moved farther away from the tablet andto the left, the application may change the work mode to zoom out todisplay a larger context for the painting operations in progresscentered on an area to the left of the area previously centered in thedisplay. In other embodiments, when zooming out using an auto zoomfeature, the application may display the entire image that is beingcreated and/or edited, regardless of any panning motion made by thestylus.

If an auto zoom feature is not enabled for the current work session,shown as the negative exit from 1160, the method may include theapplication changing the work mode to pan according to the stylusmotion, as in 1170. For example, if the stylus moved up and to theright, the application may change the work mode such that the display iscentered on an area that is above and to the right of the areapreviously centered in the display in order to view and/or performediting work on the portion of the image in that area. If the stylusmoved down and to the left, the application may change the work modesuch that the display is centered on an area that is below and to theleft of the area previously centered in the display in order to viewand/or perform editing work on the portion of the image in that area.

The proximity-based zooming and panning operations described above maybe further illustrated by way of the example displays depicted in FIGS.12A-12B and FIGS. 13A-13B. These figures illustrate an image beingedited in a natural media painting application, according to oneembodiment. FIG. 12A illustrates a display 200 of a natural mediapainting application in which an image being edited is displayed in workarea 208. In this example, a stylus representing brush tool 1200 ismoved closer to a tablet input device, such as tablet 130 describedherein, while the application is in an auto zoom mode. In response tothe stylus movement, the application zooms in on the area around brushtool 1200, as shown in FIG. 12B. FIG. 13A illustrates a display 200 of anatural media painting application in which an image being edited isdisplayed in work area 208. In this example, a stylus representing brushtool 1300 is moved to the right (as indicated by the dashed arrow pointto the right) while hovering over (and while in proximity to) a tabletinput device, such as tablet 130 described herein, while the applicationis in an auto pan mode. In response to the stylus movement, theapplication pans to the right of the area over which where brush tool1300 was displayed, and the resulting display is shown in FIG. 13B.

At least some tablets have a touch-sensitive surface in addition to theability to detect stylus manipulations. Some embodiments may use atouch-and-stylus combination gesture to provide one or more functions ormodes. In some embodiments, two or more such functions or modes may beprovided as options via a user interface so that the user may selectamong the two or more functions or modes enabled by the touch-and-stylusgesture. Modes provided via the touch-and-stylus gesture may include,but are not limited to, zoom mode, pan mode, steady detail work mode,and snap to grid mode.

One embodiment of a method for performing various actions in a naturalmedia painting application using touch-and-stylus combination gesturesis illustrated in FIG. 14. As illustrated in this example, the methodmay include the application detecting a touch on a tablet type inputdevice, as in 1410. For example, the tablet may be configured to detectthe position of and the force applied by a stylus, fingertip,fingernail, hand or other object that touches the tablet device. Asillustrated at 1420 in FIG. 14, the method may include the applicationdetecting (and recognizing) a stylus gesture that is performedcoincident with the detected touch. For example, a tablet/stylus inputmodule 122 of graphics application 120 illustrated in FIG. 1 may beconfigured to detect a variety of stylus gestures including, but notlimited to, those described herein.

In response to detecting a touch on a tablet coincident with detecting adetecting a stylus gesture, the application may determine an action tobe taken, or a work mode change, based on the combination of thedetected touch and the detected stylus gesture, as in 1430. For example,in some embodiments a touch-and-stylus gesture may be used to invoke achange in the work mode, such as an automatic zoom mode, automaticpanning mode, steady detail work mode, or snap to grid mode, or atouch-and-stylus gesture may trigger adaptation of one or moreparameters of the application to assist a user in performing a detectedpainting operation in progress. As illustrated at 1440 in FIG. 14, themethod may include the application performing the determined action orthe determined work mode change.

The touch-and-stylus gesture involves the user touching the surface ofthe tablet with a fingertip, knuckle, fingernail, etc. When the positionof the stylus is detected to be in proximity to the tablet, the functionprovided by the selected mode (or a default mode) may be performed. Forexample, a user may use this touch-and-stylus gesture to zoom in or outon an image or pan the image. Another mode that may be provided is a“steady detail work” mode that dampers user motions of the stylus toprovide steadier, smoother input. Examples of two such touch-and-stylusgestures are illustrated in FIGS. 15A and 15B. FIG. 15A illustrates acombination gesture in which a user is touching the surface of a tabletwith the side of his hand (e.g. to steady his hand) while making agesture with the stylus above the surface of the tablet. In oneembodiment, such a combination gesture may be used to invoke a “steadydetail work” mode, as described above. FIG. 15B illustrates acombination gesture in which a user is touching the surface of a tabletwith the tip of his finger while making a gesture with the stylus abovethe surface of the tablet. In one embodiment, such a combination gesturemay be used to invoke a snap to grid mode.

In some embodiments, in response to a touch-and-stylus gesture, anatural media painting application may recognize a particular paintingoperation in progress and may adapt or adjust one or more work flow,environment, and/or user input parameters in order to assist the user incompleting the painting operation. For example, if a tablet-stylus inputmodule 122 of a graphics application 120 detects and recognizes acombination gesture in which a user steadies his hand by touching theside of his hand to the tablet while moving the stylus steadily in onedirection (as if to draw a straight line), the application mayresponsively adjust a damping factor of the user input mechanism, enablea snap to grid mode, or invoke a line tool in order to assist the userin drawing a straight line. Similarly, if a tablet/stylus input module122 of a graphics application 120 detects and recognizes a combinationgesture in which a user steadies his hand by touching his fingernail tothe tablet while moving the stylus in a fanning motion (as if to draw anarc), the application may responsively adjust a damping factor of theuser input mechanism, disable a snap to grid mode, or invoke an arc toolin order to assist the user in drawing an arc. In another example,tablet/stylus input module 122 of a graphics application 120 may detectand recognize a combination gesture that suggests the user is tracing animage, and the application may enable an edge detection function toassist the user in tracing the image.

One embodiment of a method for using a touch-and-stylus gesture toassist a user in performing a painting operation in a natural mediapainting application is illustrated in FIG. 16. As illustrated in thisexample, the method may include the application detecting a touch on atablet type input device, as in 1610. For example, the tablet may beconfigured to detect the position of and the force applied by a stylus,fingertip, fingernail, hand or other object that touches the tabletdevice, and to convey that information to a tablet/stylus input module,such as tablet/stylus input module 122 of graphics application 120. Asillustrated at 1620 in FIG. 16, the method may include the applicationdetecting (and recognizing) a stylus gesture performed coincident withthe detected touch. For example, a tablet/stylus input module 122 ofgraphics application 120 illustrated in FIG. 1 may be configured todetect a variety of stylus gestures including, but not limited to, thosedescribed herein. In other embodiments, a touch-and-stylus combinationgesture may include a user touch in combination with a motion made bythe stylus while the stylus is also touching the tablet. In suchembodiments, in addition to detecting the motion of the stylus (e.g.,through various sensors of the tablet or via a motion detector and/oraccelerometer of the stylus), the combination gesture may be furtherdefined by detecting the pressure applied to the tablet by the stylusduring the gesture.

In response to detecting a touch on a tablet coincident with detecting adetecting a stylus gesture, the application may determine that aparticular painting function is in progress, based on the combination ofthe detected touch and the detected stylus gesture, as in 1630. Forexample, the combination gesture may suggest that the user is drawing aline or an arc, or is tracing an image, as described herein. Asillustrated at 1640 in FIG. 16, the method may include the applicationadapting one or more of its user input mechanisms (or parametersthereof) in order to assist the user in performing the detected paintingfunction in progress.

Examples of two such touch-and-stylus gestures that may be used toassist a user in performing a painting operation are illustrated inFIGS. 17A and 17B. FIG. 17A illustrates a combination gesture in which auser is touching the surface of a tablet with his fingertip while movingthe stylus steadily along the tablet in one direction (as if to draw astraight line). As described above, a natural media painting applicationthat implements the methods described herein may responsively adjustvarious parameters of a work mode or user input mechanism in order toassist the user in drawing the straight line. FIG. 17B illustrates acombination gesture in which a user is touching the surface of a tabletwith his fingertip while moving the stylus along the tablet in a fanningmotion (as if to draw an arc). In some embodiments, the natural mediapainting application described herein may responsively adjust variousparameters of a work mode or user input mechanism in order to assist theuser in drawing the arc.

Some embodiments may include a means for detecting gestures made using astylus, a tablet type input device, and/or a combination of a stylus anda tablet type input device. For example, a tablet/stylus input modulemay present an interface through which various gestures representingactions to be taken in a natural media painting application (e.g., modechanges and/or painting operations) may be detected (e.g., usingcollected motion information, pressure data, etc.) and recognized, andmay generate and store data representing the detected gestures for usein various image editing operations in the natural media paintingapplication, as described herein. The tablet/stylus input module may insome embodiments be implemented by a non-transitory, computer-readablestorage medium and one or more processors (e.g., CPUs and/or GPUs) of acomputing apparatus. The computer-readable storage medium may storeprogram instructions executable by the one or more processors to causethe computing apparatus to perform presenting an interface through whichvarious gestures may be detected and recognized, detecting andrecognizing those gestures, and generating and storing data representingthose gestures for subsequent use in the natural media paintingapplication, as described herein. Other embodiments of the tablet/stylusinput module may be at least partially implemented by hardware circuitryand/or firmware stored, for example, in a non-volatile memory.

Some embodiments may include a means for mapping detected gestures madeusing a stylus and/or tablet type input device to various functions of anatural media painting application. For example, a gesture mappingmodule may receive input specifying various gestures that have beendetected, may determine actions to be taken in a natural media paintingapplication (e.g., image editing operations to be performed using abrush tool in the application, or mode changes to be applied in theapplication) in response to that input, and may generate and store datarepresenting the actions to be taken in the natural media paintingapplication, as described herein. The gesture mapping module may in someembodiments be implemented by a non-transitory, computer-readablestorage medium and one or more processors (e.g., CPUs and/or GPUs) of acomputing apparatus. The computer-readable storage medium may storeprogram instructions executable by the one or more processors to causethe computing apparatus to perform receiving input specifying variousgestures that have been detected, determining actions to be taken in anatural media painting application in response to that input, andgenerating and storing data representing the actions to be taken in thenatural media painting application, as described herein. Otherembodiments of the gesture mapping module may be at least partiallyimplemented by hardware circuitry and/or firmware stored, for example,in a non-volatile memory.

Some embodiments may include a means for simulating the behavior of abristle brush in a natural media painting application. For example, apainting simulation module (which may include a bristle brush model),may receive input specifying various painting actions to be performed ina natural media painting application (e.g., image editing operations tobe performed using a brush tool in the application) in response totablet and/or stylus input, and may generate and store data representingan image that has been modified by the various image editing operationsin the natural media painting application, as described herein. Thepainting simulation module may in some embodiments be implemented by anon-transitory, computer-readable storage medium and one or moreprocessors (e.g., CPUs and/or GPUs) of a computing apparatus. Thecomputer-readable storage medium may store program instructionsexecutable by the one or more processors to cause the computingapparatus to perform receiving input specifying various painting actionsto be performed in a natural media painting application (e.g., imageediting operations to be performed using a brush tool in theapplication) in response to tablet and/or stylus input, and generatingand storing data representing an image that has been modified by thevarious image editing operations in the natural media paintingapplication, as described herein. Other embodiments of the paintingsimulation module may be at least partially implemented by hardwarecircuitry and/or firmware stored, for example, in a non-volatile memory.

Example Computer System

The methods illustrated and described herein may be executed on one ormore computer systems, which may interact with other devices, accordingto various embodiments. One such computer system is illustrated in FIG.18. In the illustrated embodiment, computer system 1900 includes one ormore processors 1910 coupled to a system memory 1920 via an input/output(I/O) interface 1930. Computer system 1900 further includes a networkinterface 1940 coupled to I/O interface 1930, and one or moreinput/output devices 1950, such as cursor control device 1960, keyboard1970, audio device 1990, and display(s) 1980. Input/output devices 1950include a tablet 130 and stylus 140 for enabling natural media paintingusing a realistic brush and tablet stylus gestures as described herein.In some embodiments, it is contemplated that embodiments may beimplemented using a single instance of computer system 1900, while inother embodiments multiple such systems, or multiple nodes making upcomputer system 1900, may be configured to host different portions orinstances of embodiments. For example, in one embodiment some elementsmay be implemented via one or more nodes of computer system 1900 thatare distinct from those nodes implementing other elements.

In various embodiments, computer system 1900 may be a uniprocessorsystem including one processor 1910, or a multiprocessor systemincluding several processors 1910 (e.g., two, four, eight, or anothersuitable number). Processors 1910 may be any suitable processor capableof executing instructions. For example, in various embodiments,processors 1910 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 1910 may commonly,but not necessarily, implement the same ISA.

In some embodiments, at least one processor 1910 may be a graphicsprocessing unit. A graphics processing unit or GPU may be considered adedicated graphics-rendering device for a personal computer,workstation, game console or other computer system. Modern GPUs may bevery efficient at manipulating and displaying computer graphics, andtheir highly parallel structure may make them more effective thantypical CPUs for a range of complex graphical algorithms. For example, agraphics processor may implement a number of graphics primitiveoperations in a way that makes executing them much faster than drawingdirectly to the screen with a host central processing unit (CPU). Invarious embodiments, the methods as illustrated and described in theaccompanying description may be implemented by program instructionsconfigured for execution on one of, or parallel execution on two or moreof, such GPUs. The GPU(s) may implement one or more applicationprogrammer interfaces (APIs) that permit programmers to invoke thefunctionality of the GPU(s). Suitable GPUs may be commercially availablefrom vendors such as NVIDIA Corporation, ATI Technologies, and others.

System memory 1920 may be configured to store program instructionsand/or data accessible by processor 1910. In various embodiments, systemmemory 1920 may be implemented using any suitable memory technology,such as static random access memory (SRAM), synchronous dynamic RAM(SDRAM), nonvolatile/Flash-type memory, or any other type of memory. Inthe illustrated embodiment, program instructions and data implementingdesired functions, such as those for methods as illustrated anddescribed in the accompanying description, are shown stored withinsystem memory 1920 as program instructions 1925 and data storage 1935,respectively. In other embodiments, program instructions and/or data maybe received, sent or stored upon different types of computer-accessiblemedia or on similar media separate from system memory 1920 or computersystem 1900. Generally speaking, a computer-accessible medium mayinclude storage media or memory media such as magnetic or optical media,e.g., disk or CD/DVD-ROM coupled to computer system 1900 via I/Ointerface 1930. Program instructions and data stored via acomputer-accessible medium may be transmitted by transmission media orsignals such as electrical, electromagnetic, or digital signals, whichmay be conveyed via a communication medium such as a network and/or awireless link, such as may be implemented via network interface 1940.

In one embodiment, I/O interface 1930 may be configured to coordinateI/O traffic between processor 1910, system memory 1920, and anyperipheral devices in the device, including network interface 1940 orother peripheral interfaces, such as input/output devices 1950,including tablet 130 and stylus 140. In some embodiments, I/O interface1930 may perform any necessary protocol, timing or other datatransformations to convert data signals from one component (e.g., systemmemory 1920) into a format suitable for use by another component (e.g.,processor 1910). In some embodiments, I/O interface 1930 may includesupport for devices attached through various types of peripheral buses,such as a variant of the Peripheral Component Interconnect (PCI) busstandard or the Universal Serial Bus (USB) standard, for example. Insome embodiments, the function of I/O interface 1930 may be split intotwo or more separate components, such as a north bridge and a southbridge, for example. In addition, in some embodiments some or all of thefunctionality of I/O interface 1930, such as an interface to systemmemory 1920, may be incorporated directly into processor 1910.

Network interface 1940 may be configured to allow data to be exchangedbetween computer system 1900 and other devices attached to a network,such as other computer systems, or between nodes of computer system1900. In various embodiments, network interface 1940 may supportcommunication via wired or wireless general data networks, such as anysuitable type of Ethernet network, for example; viatelecommunications/telephony networks such as analog voice networks ordigital fiber communications networks; via storage area networks such asFibre Channel SANs, or via any other suitable type of network and/orprotocol.

Input/output devices 1950 may, in some embodiments, include one or moredisplay terminals, keyboards, keypads, touchpads, scanning devices,voice or optical recognition devices, tablets and stylus, or any otherdevices suitable for entering or retrieving data by one or more computersystem 1900. Multiple input/output devices 1950 may be present incomputer system 1900 or may be distributed on various nodes of computersystem 1900. In some embodiments, similar input/output devices may beseparate from computer system 1900 and may interact with one or morenodes of computer system 1900 through a wired or wireless connection,such as over network interface 1940.

As shown in FIG. 18, memory 1920 may include program instructions 1925,configured to implement embodiments of methods as illustrated anddescribed in the accompanying description, and data storage 1935,comprising various data accessible by program instructions 1925. In oneembodiment, program instructions 1925 may include software elements ofmethods as illustrated and described in the accompanying description,including a tablet/stylus input module, painting simulation module,bristle brush model, and/or gesture mapping module. Data storage 1935may include data that may be used by these and other modules in someembodiments. In other embodiments, other or different software elementsand/or data may be included in memory 1920.

Those skilled in the art will appreciate that computer system 1900 ismerely illustrative and is not intended to limit the scope of methods asillustrated and described in the accompanying description. Inparticular, the computer system and devices may include any combinationof hardware or software that can perform the indicated functions,including computers, network devices, internet appliances, PDAs,wireless phones, pagers, etc. Computer system 1900 may also be connectedto other devices that are not illustrated, or instead may operate as astand-alone system. In addition, the functionality provided by theillustrated components may in some embodiments be combined in fewercomponents or distributed in additional components. Similarly, in someembodiments, the functionality of some of the illustrated components maynot be provided and/or other additional functionality may be available.

Those skilled in the art will also appreciate that, while various itemsare illustrated as being stored in memory or on storage while beingused, these items or portions of them may be transferred between memoryand other storage devices for purposes of memory management and dataintegrity. Alternatively, in other embodiments some or all of thesoftware components may execute in memory on another device andcommunicate with the illustrated computer system via inter-computercommunication. Some or all of the system components or data structuresmay also be stored (e.g., as instructions or structured data) on acomputer-accessible medium or a portable article to be read by anappropriate drive, various examples of which are described above. Insome embodiments, instructions stored on a computer-accessible mediumseparate from computer system 1900 may be transmitted to computer system1900 via transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as a network and/or a wireless link. Various embodiments mayfurther include receiving, sending or storing instructions and/or dataimplemented in accordance with the foregoing description upon acomputer-accessible medium. Accordingly, the systems and methodsdescribed herein may be practiced with other computer systemconfigurations.

Various embodiments may further include receiving, sending or storinginstructions and/or data implemented in accordance with the foregoingdescription upon a computer-accessible medium. Generally speaking, acomputer-accessible medium may include storage media or memory mediasuch as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile ornon-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.),ROM, etc., as well as transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as network and/or a wireless link.

The various methods as illustrated in the figures and described hereinrepresent examples of embodiments of methods. The methods may beimplemented in software, hardware, or a combination thereof. The orderof method may be changed, and various elements may be added, reordered,combined, omitted, modified, etc. Various modifications and changes maybe made as would be obvious to a person skilled in the art having thebenefit of this disclosure. It is intended that the disclosure embraceall such modifications and changes and, accordingly, the abovedescription to be regarded in an illustrative rather than a restrictivesense.

What is claimed is:
 1. A method, comprising: using a computer toperform: collecting information about a user manipulation of a stylus inrelation to a tablet device associated with the computer; recognizing,from the collected information, a stylus gesture performed by the uservia manipulation of the stylus, such that: the stylus gesture is one ofa plurality of stylus gestures that are recognized by the computer toperform at least one of a plurality of actions in a graphics applicationthat comprises a natural media painting application, at least some ofthe stylus gestures are mapped to user manipulation of the stylus at adistance from the tablet device, at least some of the stylus gesturesinvolve contact of the stylus with the tablet device, and the stylusgestures include a brush switching gesture in which a proximity of thestylus to the tablet changes from a first position relative to thetablet to a second position that is further away from the tablet andwith at least a rate of change that corresponds to the brush switchinggesture, the first position being within a first pre-defined distancethreshold relative the tablet and the second position being beyond asecond pre-defined distance threshold, and switching betweenpaintbrushes of a brush tool being performed responsive to recognitionof the brush switching gesture; determining which of the plurality ofactions to perform based on the recognized stylus gesture; andperforming a painting function for a digital image in the graphicsapplication including performing the determined actions.
 2. The methodof claim 1, wherein the tablet device is an integrated component of thecomputer.
 3. The method of claim 1, wherein the collected informationcomprises information about pressure applied to the tablet device by thestylus during the manipulation of the stylus.
 4. The method of claim 1,wherein the plurality of stylus gestures that are recognized by thecomputer includes a translation in one or more dimensions, a twistingmotion, a barrel rotation, a fanning motion, a mashing down of thestylus into a tablet device, a jerking down motion, shaking the stylusaway from the tablet device, and shaking the stylus toward the tabletdevice.
 5. The method of claim 1, wherein the plurality of actionscomprises one or more of: sharpening a point of a brush tool, cleaningthe brush tool, refilling the brush tool, changing a tool, drying thebrush tool, splattering paint from the brush tool, adding paint to thebrush tool saturating the brush tool, homogenizing the colors on the tipof the brush tool, or splitting the tip of the brush tool in thegraphics application.
 6. The method of claim 1, wherein the collectedinformation comprises one or more of: spatial or directional informationcollected during the manipulation of the stylus, acceleration datacollected during the manipulation of the stylus, the first position ofthe stylus, the second position of the stylus, an initial orientation ofthe stylus, or an ending orientation of the stylus.
 7. The method ofclaim 1, wherein the determined actions are determined based, at leastin part, on a context in which the user manipulation of the stylus wasperformed.
 8. The method of claim 1, wherein the plurality of stylusgestures includes shaking the stylus towards the tablet, shaking thestylus away from the tablet, and twisting the stylus about the majoraxis of the stylus; and wherein the action that is performed in responseto recognition of shaking the stylus towards the tablet includessplattering paint; wherein the action that is performed in response torecognition of shaking the stylus away from the tablet includes cleaningthe brush tool; and wherein the action that is performed in response torecognition of twisting the stylus includes homogenizing the colors ofpaint on the brush tool.
 9. The method of claim 1, wherein the pluralityof stylus gestures includes waving the stylus back and forth above thetablet, mashing the stylus into the tablet, and twisting the stylusabout the major axis of the stylus; and wherein the action that isperformed in response to recognition of waving the stylus back and forthabove the tablet includes drying the brush tool; wherein the action thatis performed in response to recognition of mashing the stylus into thetablet includes splitting the bristles of the brush tool; and whereinthe action that is performed in response to recognition of twisting thestylus includes sharpening the point of the brush tool.
 10. The methodof claim 1, wherein the collected information includes acceleration datathat is collected using an accelerometer of the stylus to augment othercollected information, the acceleration data being usable to determinethe rate of change for proximity changes of the stylus between the firstposition and the second position.
 11. A system, comprising: a stylusinput module; a display device; and a computer device that duringoperation executes the stylus input module to perform operationscomprising: collecting information about a user manipulation of a stylusin relation to the computer device for both manipulation at a distancefrom the computer device and manipulation to provide touch input, theinformation including at least information regarding motion,orientation, and position of the stylus and an amount of pressureapplied with the stylus; recognizing, from the collected information, astylus gesture performed by the user via manipulation of the stylus,wherein the stylus gesture is one of a plurality of stylus gestures thatare recognized by the computer device to perform at least one of aplurality of actions in a graphics application, including changing abrush type; determining which of the plurality of actions to performbased on the recognized stylus gesture; wherein the changing the brushtype is determined in response to recognition of a brush switchingstylus gesture in which a proximity of the stylus changes from a firstposition relative to the display device to a second position that isfurther away from the display device and with at least a rate of changethat corresponds to the brush switching stylus gesture, the rate ofchange determined using acceleration data that is collected from anaccelerometer of the stylus to augment the collected information andindicates acceleration that results from jerking the stylus to changethe proximity; and performing a painting function for a digital imagedisplayed on the display device by the graphics application includingperforming the determined actions.
 12. The system of claim 11, furthercomprising: a pressure sensitive surface; wherein the collectedinformation comprises information about the manipulation of the stylusin relation to the pressure sensitive surface.
 13. The system of claim11, wherein the graphics application comprises a natural media paintingapplication, and wherein the painting function comprises a simulation ofa real-world painting action using a brush tool.
 14. The system of claim11, wherein the collected information comprises one or more of: spatialinformation collected during the manipulation of the stylus, directionalinformation collected during the manipulation of the stylus, theacceleration data collected during the manipulation of the stylus, thefirst position of the stylus, the second position of the stylus, aninitial orientation of the stylus, or an ending orientation of thestylus.
 15. The system of claim 11, wherein the plurality of stylusgestures includes a barrel rotation, a fanning motion, mashing down thestylus, a jerking down motion, shaking the stylus away from a tabletdevice, and shaking the stylus toward the tablet.
 16. A non-transitorycomputer-readable storage medium storing program instructions that whenexecuted on one or more computers cause the one or more computers toperform acts comprising: collecting information about a usermanipulation of a stylus indicative of six degrees of freedom movementof the stylus relative to an initial position of the stylus inthree-dimensional space relative to a touch input surface andinformation regarding touch or pressure input applied to the touch inputsurface; recognizing, from the collected information, a stylus gestureperformed by the user via manipulation of the stylus, wherein the stylusgesture is one of a plurality of stylus gestures that are recognized bythe one or more computers to perform at least one of a plurality ofactions in a graphics application, and the plurality of stylus gesturesincludes a twisting motion, a waving motion above the touch inputsurface, a brush switching gesture in which a proximity of the stylusrelative the touch input surface is changed with at least a rate ofchange that corresponds to the brush switching gesture and changes fromthe initial position relative the touch input surface that is within afirst pre-defined distance threshold of the touch input surface to anending position that is further away from the touch input surface andbeyond a second pre-defined distance threshold, a quick motion towardthe touch input surface, shaking the stylus directed away from the touchinput surface, or shaking the stylus directed towards the touch inputsurface; determining which of the plurality of actions to perform basedon the recognized stylus gesture, wherein an action of switching betweenpaintbrushes of a brush tool is performed in response to recognition ofthe brush switching gesture; and performing a painting function for adigital image in the graphics application including performing thedetermined actions.
 17. The storage medium of claim 16, wherein thecollected information comprises information about the manipulation ofthe stylus in relation to a tablet device having the touch inputsurface.
 18. The storage medium of claim 16, wherein the graphicsapplication comprises a natural media painting application, and whereinthe painting function comprises a simulation of a real-world paintingaction using the brush tool.
 19. The storage medium of claim 16, whereinthe collected information comprises one or more of: spatial informationcollected during the manipulation of the stylus, directional informationcollected during the manipulation of the stylus, acceleration datacollected during the manipulation of the stylus, the initial position ofthe stylus, the ending position of the stylus, an initial orientation ofthe stylus, or an ending orientation of the stylus.
 20. The storagemedium of claim 16, wherein the plurality of stylus gestures furtherincludes mashing down the stylus.